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Abstract 

A major goal of the Space Station Era is to reduce reliance on support 
from ground based experts. The development of software programs using 
expert systems technology is one means of reaching this goal without 
requiring crew members to become intimately familiar with the many 
complex spacecraft subsystems. Development of an expert systems program 
requires a validation of the software with actual flight hardware. By 
combining accurate hardware and software modelling techniques with a 
modular systems approach to expert systems development, the validation of 
these software programs can be successfully completed with minimum risk 
and effort. The TIMES Expert System (TES) is an application that monitors 
and evaluates real-time data to perform fault detection and fault isolation 
tasks as they would otherwise be carried out by a knowledgeable designer. 
This paper discusses (1) the development process and primary features of 
TES, (2) a modular systems approach and (3) lessons learned. 


Introduction 

The Thermoelectric Integrated Membrane Evaporation Subsystem 
(TIMES) is a spacecraft life support system designed and produced by 
Hamilton Standard . The TIMES Expert System (TES) is a prototype developed 
as a first step toward capturing in-house diagnostic expertise on this 
subsystem. The primary goals of the prototype development effort were to 
(1) investigate expert systems as space-based tools for reducing reliance on 
ground based support and (2) explore a modular systems approach to 
developing real-time expert systems. To address these goals, TES was 
develppea as a fault detection and fault isolation expert system to monitor 
real-time TIMES data. The TIMES Expert System was created under 
NASA/Johnson Space Flight Center contract number NAS 9-17209. 


What is TIMES? 


TIMES is a spacecraft 
waste water processor that 
employs vacuum distillation, 
thermoelectric heat pumping, 
and membrane separation to 
reclaim high quality product 
water [2j. A system 
controller contains the 
electrical and software logic 
necessary for process 
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control, instrumentation and critical fault detection. In addition to data 
provided to a MS-1553B bus, sensor readings and performance calculations 
are made available by the controller via RS-232C tor use by data checkout 
and recording systems. An electrical driver box standardizes all signals 
between the hardware and the controller. The TIMES, along with numerous 
other life support subsystems, is operated and visually monitored from a 
central Display and Control Console (DCC) via a MS-1553B bus (see Figure 1). 


Primary TES Features 

The TIMES Expert System was developed using Knowledge Engineering 
Environment (KEE ) and Common Lisp on a Symbolics computer. 
Approximately 45 potential subsystem and sensor problems have been 
addressed as both independent and concurrent failures. The TIMES controller 
relies on threshold violations to provide limited error detection and will 
shut the subsystem down if a critical problem is suspected. TES enhances 
controller capabilities by operating as an early warning system, providing 
additional fault tolerance ana acting as an extended problem explanation tool 
for the astronaut. 

Combining the graphics features of KEE and Common Lisp has given TES 
versatile display capabilities. Some examples are dynamic flows, valve 
positions, and sensor readings presented on an operational schematic of the 
TIMES. In addition to these features, the operator also has the ability to 
customize the monitoring environment by replacing some or all numerical 
displays with dials and to view real-time historical trend plots of expected 
and actual values for all sensor and performance data. 


A Modular System Design 

When real-time prototype hardware output results in inputs to an expert 
system, the following key issues must be addressed: 

1. Limited availability of system hardware for expert system 
testing and verification due to: 

* A limited number of each prototype system is 
usually developed and available. 

* Prototype systems are often dedicated to 
rigorous test schedules and numerous design changes. 

* System testing often has priority over expert 
system testing. 

* It is costly and difficult to move hardware from one 
site to another for demonstration or testing 
purposes. 

2. Actual insertion of faults into prototype hardware is 
unrealistic in many cases because: 

* Altering a system component to introduce a fault 
can be costly and sometimes hazardous. 

* Many system sensors and components are not 
physically accessible for alteration. 

These issues may be addressed by taking a modular approach to system 
design. The modular approach involves the development of an accurate 
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software simulation of the hardware (See Figure 2). The closer the model is 
to duplicating the hardware's data output in terms of content, format and 
protocol, the more valuable it will be when addressing the above issues. 



Figure 2: Modular Real Time Testing 


Modularity has been applied 
to the TES program by using a 
thermodynamic model of TIMES as 
the expert system test data 
source. The TIMES model was 
developed using Quick BASIC on an 
IBM PC. The RS-232C output from 
this model realistically duplicates 
the RS-232C test data output from 
the TIMES controller in content, 
format and protocol. This data 
supplies TES with the sensor, 
control and performance 
information it needs to monitor 
the TIMES (model). 


The TIMES model can be operated as a stand alone system or in 
conjunction with the TIMES Expert System. The model's graphics interface 
allow the user to visually monitor simulated normal TIMES operation, or to 
insert faults and observe the effects on subsystem operation independent of 
the TIMES Expert System. This 
allows for independent model 
verification using TIMES data and 
makes the model itself a valuable 
tool in understanding detailed 
TIMES performance and fault 
characteristics. The model, with 
its interface (Figure 3), 
effectively replaces the TIMES 
hardware and the DCC (Figure 1) 
for testing and demonstration of 
TES. 



Figure 3: The TIMES Expert System 


The modular approach taken in the TES design has helped to reduce 
development costs by providing a more flexible and accessible testing and 
verification platform. Fxir example, TIMES operates on an approximate 24 
hour cycle and fault symptoms often vary as a function of the cycle duration. 
To test the TES fault detection capabilities and insure that cyclic data is 
being handled successfully, testing must be conducted at numerous cycle 
points. Tests must be repeated a number of times because expert system 
evaluation is an iterative process [4] . These test constraints would demand 
valuable subsystem time and may require costly repair or replacement of 
components damaged during testing or additional hardware to simulate 
failures. By using a software model as the test bed, an open circuit in a 
thermoelectric module or a separator motor magnetic drive decoupling can be 
generated repeatedly. Simulation rates can be adjusted to decrease test 
time. In these ways, the TIMES model facilitates testing without damaging 
subsystem hardware or interrupting hardware testing. 
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A Modular Diagnostic Approach 

Remaining consistent with a modular systems approach, TES's 
diagnostic content is also structured modularly. Application of verbal 
reporting techniques [3] during the formalization phase [4] of TES's 
development revealed a distinctly two level diagnostic approach used by the 
experts. High level performance parameters provide the first clues to a 
subsystem performance problem. Detailed diagnostics are only employed 
when such a problem is detected. Because it is the function of the TIMES to 
efficiently reclaim high quality product water, there are three primary 
indicators of degraded performance: (1) low water production rate, (2) poor 
product water quality, and (3) high power consumption. The first two areas 
have been addressed by the TES prototype. This approach has been captured 
in a modular fashion modelled after that of the experts allowing for rapid 
expert system performance, and facilitating future expansion to include 
other systems [1], [5]. 


A front end processor in TES 
monitors TIMES performance 
parameters using trend analysis to 
watch for significant indication of 
a problem. During normal 

operation, the only other 
functions performed by TES are the 
storing of trend data, real-time 
display update and periodic sensor 
checkout to flag drifts or 


High Level Health Parameter Monitor 



Figure 4: TES Functional Modularity 


inconsistent readings. Only when subsystem performance appears to have 
deteriorated does TES perform detailed diagnosis to isolate the problem and 
warn the astronaut before shutdown thresholds are exceeded. (Figure 4) 


The rules in the TIMES expert system have been divided into four major 
categories: (1) General Health rules, (2) Water Production Rate rules, (3 
Water Quality rules and (4) Sensor Health rules. Division of the rule base 
into problem areas has further enabled rapid fault diagnosis. For instance, 
if a low water production rate were detected, TES would reason over rule 
types (1) and (2) drawing on the current knowledge base and trend 
information to diagnose the problem. A separate Sensor Health rule base has 
increased TES credibility by flagging unreliable sensors so that they are not 
relied on in future reasoning, and by differentiating between sensor drifts or 
failures and actual subsystem health problems (i.e. high temperature). 


Conclusions 

Using the TIMES model as an input to the TIMES Expert System provides 
a flexible and portable means of addressing the availability, cost, and time 
issues associated with developing an expert system to monitor real-time 
hardware data. It also increases TES's diagnostic credibility by separating 
fault insertion and fault detection. The operator can observe TIMES sensor 
readings and performance indicators on the model to reach his own 
conclusions, while the expert system provides the expert diagnosis. 

TES's, functional modularity, patterned after the expert's own 
diagnostic approach, increases its monitoring efficiency without limiting 
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future growth. The front end processor makes TES an efficient, high level 

health monitor until detailed anomaly analysis is needed. The front end 
processor also supplies TES (and hence the operator) with a trend history of 
all sensors and performance parameters. Detailed trend analysis on this data 
supplies the TES reasoning system with the information that the expert uses 
by either visually examining trend plots or performing detailed analysis of 
historical data. 

Independent sensor monitoring increases TES's diagnostic capability by 
flagging unreliable sensors thus excluding them from use in future reasoning 
ana by distinguishing between sensor drift or failure and actual performance 
problems. 

Future TES expansion areas could include: (1) further development of 
the front end processor to address more complex data characteristics, (2) a 
detailed feasibility study that would address the modular system design to 
multiple systems, (3) connection to TIMES hardware, and (4) further 
investigation of multiple faults. 
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